NAME OF INVENTION AND BRIEF DESCRIPTION: 


at 


Game Learning and Playing Machine: 


Machine, in the course of playing a series of games with a human, 
gradually builds up a "dictionary" of winning moves, by applica~ 
tion of the rules of the game and the strategy of seeking a move 
that will prevent human from being able to make a winning move. 
In the process, machine commits, but later corrects, errors. 


The machine plays a "counting game" ~- wherein players alternately 
choose numbers (in this case, from 1 to 6) restricted by certain 
rules (in this case, neither may play the number just played by 
opponent nor its complement with respect to 7) and each attempts 
to make the combined total of all plays equal a certain number 
exactly at the end of his play. In model the goal is 18. 


This type game is characterized by existence, at each "total" 
point,of certain "winning" plays for player whose turn it is to 
play. Machine, on its turn, looks ahead in its memory, trying 
to find a play that will result in opponent being unable to make 
a winning play. Plays that meet this criterion are "marked down" 
as. that "total point" as "winning". Any play so (erroneously) 
marked that would permit opponent to make a winning play on his 
(next) turn, is "erased". 


WHAT IS NEW AND NOVEL IN THIS INVENTION OVER THE PRIOR ART? 


(1) Machine builds up "memory" of winning moves gradually during 
a sequence of games. Other game players have a prewired or 
fixed dictionary of winning moves. | 

(2) Machine records wrong information in its memory and later 
corrects it. 

(3) By virtue of choice of game, machine is much simpler than 
others that learn other games that are more complex to 
mechanize. 

(4) Selects losing plays at random. 

(5) State of "learning" is displayed on front panel, with "good" 
information learned distinguished from "bad". (A variation: 
memory display on front panel in same arrangement as that 
inside,to "help" human - of course he then is not told which 
is "good" ~ which "bad". ) 

(6) Machine occasionally "goofs", i.e. chooses a play at random 
instead of seeking a winning play; this gives human a chance 
to win even when machine's "dictionary" of winning moves is 
complete and accurate. 


CONCEPTION OF IDEA: 


Conception Dates: 


(1) Basic idea of machine to play this type of game: 27 May 1953 

(2) Simulated learning by connecting in portions of prewired 
memory — depending on number of games won by human: 
20 April 1954 

(3) Memory scanner for present learning technique (at that time 
I concluded that it would not work): 14 September 1954 

(4) Realization that if machine could erase as well as write into 
memory, idea (3) above would work and machine would gradually 
build up correct memory: between 1 and rope en en 1958 
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(5) Random choice of "losing" play a réad into memory both 
before and after machine play is added to total - as needed 
to combat skillful human play: 12 October 1959 

(6) Goof circuit: 5 November 1959 

(7) Necessity to disable memory cell at column 5, row 2; hecause 
of inability of machine to find winning response to opening 
play of 1, if error is recorded in column 7: 28 November 1959 

DISCLOSURES ; 


Present learning technique was disclosed to various engineers, 
verbally, after December 1958. Among them were F. Morrison, 
B. Metteer, and M. Corwin, all engineers at WREC. 


Roy Goodman, a neighbor, also was kept informed, and saw first 
demonstration of model on 6 December 1959. 


Talk presented on complete machine 14 March 1960 to engineers 
of Data Systems Lab, WMEC. 


METHOD OF OPERATION 


Attach line cord. Operate POWER switch, S14, to ON. Operate 
MEMORY RESET push button S16 to set contents of memory to "zero". 
If desired to preset 6 winning plays near the end of the game 
into the memory, operate MEMORY PRELOAD push button 815. Operate 
NEW GAME push button 812. This button must be depressed long 
enough for motor to start memory scanner travelling towards be- 


ginning of game. 


After motor stops, indicating that memory scanner is ready at 
"0", human selects his first play by setting PLAY SELECTOR S3 to 
one of 6 positions, corresponding to choices of "1", "2", "3", 
"4", "5", or "6". Next he depresses PLAY push button S13 for 
about one-half second. Programmer stepping switch 1 then starts. 
First the programmer clears the machine play register and then 
it controls the advancing of totalizer 2 to add the human's play 
to the total, which started at 0. The programmer now waits till 
the opening of relay 19 signals that the memory scanner drive 
motor has stopped and the memory scanner is accordingly properly 
positioned so that it is looking at the next 6 memory columns. 
Then the programmer causes the machine play selector, stepping 
Switch 4, to search for a winning move. (The technique for this 
is described later. For the moment, if relay 6 is closed, it 
indicates that the move being tested is not a winning move; if 6 
is open, it is a winning move.) Interconnections between 4 and 
S3 test each move tried by the machine to see if it is legal. If 


the move being tried is a winning move, stepper 4, and relays 6 
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and 8 cooperate to 'mark it in the memory" as a winning move; if 

it is a losing move, they act to erase any mistakenly recorded 
indication that it is a winning move. Selector 4 searches in 

the order, ~ 1,2,3,4,5,6,5,4,3,2,1. If after this sequence no 
legal winning move is found, relay 20 is operated and locked, and 

a random selection of a losing move is initiated. Relays 24 and 

25 now substitute for relay 6, and a legal losing move is chosen 
by selector 4, which stops when it has selected a legal, randomly 
chosen move. The programmer 1 now records the selected move in 

the machine play register, consisting of relays 21, 22, and 23. 
Next it advances the totalizer, stepper 2, to add in the machine's 
play. Again the programmer waits for the memory scanner to get 
into position. Programmer 1 now causes machine play selector 4 to 
Start from the selected play and advance to its home position, 
writing winning and losing moves into its memory at the new total 
as it does so. For example, suppose the machine had just selected 
a play of 4. During this phase it would sequence 4,5,6,5,4,3,2,1 - 
accordingly all positions of the memory in the column corresponding 
to the current total would be "read into”. On the other hand, 
during the selection of a winning move, if a winning legal move 
exists, the machine reads into its memory only up to and including 
that move. In the example cited, it would have read into positions 
1, 2, 3, and 4. Had the machine been unable to find a legal 


winning move, all rows at that total would have been read into. 


The programmer 1 now proceeds and after covering some blank steps 
stops at its home position and turns on the YOUR TURN indicator. 


This completes one cycle; the human now makes another selection 
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for his second move and the cycle repeats. 


If the human selects an illegal move, relay 5 operates and dis- 
connects the programmer 1 from the clock generator output relay 11, 
thus preventing the program from starting. Relay 5 is additionally 
employed to transfer the clock generator output relay from the 
programmer 1 to the machine play selector driver relay 8 at the 


proper time in the program. 


When the totalizer 2 reaches the 18th position, at the end of 

the game, relay 10 operates. Relay 10 turns on a 115 V AC lamp 
to illuminate the number "18". It also activates a logic network 
involving relay 9 (the totalizer controller) and the programmer, 
which makes the decision as to who won the game. The programmer 
position is used to determine whose turn it was, and relay 9 is 
used to determine whether 18 was reached exactly at the end of 


counting in the play in question, or not. 


When NEW GAME push button S12 is activated, first the programmer 1 
is self-stepped to its home position, which is detected by relay 17. 
When 1 is at home, relay 17 transfers homing power to the totalizer 
stepper 2. When stepper 2 reaches home (indicating total of 0) 
relay 18 operates and is used to de-energize the machine play 
indicator lamps and to deactivate the illegal play circuit so the 
human can choose any play without regard to the play held by the 
machine play register. Meanwhile, another contact on S12 has 
operated relay 27. This relay inverts the slewing AC voltage 

that is fed to the memory scanner servo amplifier (this is ex- 


plained below) causing the scanner to be driven back towards the 
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beginning of the game - towards the "early" part of the memory . 
Relay 27 locks down until the opening of relay 19 indicates that 
the motor has stopped with the memory scanner at the 0 position 
in the memory. Contacts on relays 19 and 1% are used to disable 
the clock generator so that depressing the PLAY push button $13 
will not cause the programmer to start until the memory scanner 


is at the 0 position. 


The random losing plays are selected as follows: When machine 
play selector 4 has made one full circuit (indicating a lack of 
legal winning moves) its homing contact 4b' closes and causes 
relay 20 to lock down. Contacts of 20 close across the timing 
capacitors in two neon flip-flops that otherwise run continuously. 
This action "clamps" each flip-flop so that the neon lamp of each 
pair that was lit when 20 operated remains lit. The flip-flops 
control sensitive relays 24a and 25a, which in turn control 

relays 24 and 25. Depending on the combinations of conditions 

of being off or on, one of four states is selected. Each state 
generates a pair of selected plays as follows: Relay 24 and 25 
both operated ~- 1 or 5; relay 24 operated, 25 not - 2 or 6; relay 
24 not operated, relay 25 operated - 3 or 5; both relays not 
operated ~ 4 or 6. In spite of the fact that 5 and 6 are selected 
twice as many times as 1, 2, 3, or 4 ~ the fact that machine play 
selector 4 always selects in the order 1,2,3,4,5,6 results in there 
being an equal probability that each of the four legal plays will 


be selected. 


This feature, of selecting the losing plays at random, was found 


to be necessary for the machine to correct mistakes in its memory. 
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At first I designed the machine so that it would make a losing play 
of either 1 or 2, whichever was legal, or 1 in case both were legal. 
The trouble with this was as follows: If the machine is losing, it 
neans thet either a mistake is recorded in the nemory or a winning 
play has not been recorded. If the human capitalizes on this by 
repeatedly playing the same game, without the random play feature 
the machine will not play to the positions in its memory where the 
above-mentioned troubles exist. It will merely respond by playing 
the same sequences of moves and will not learn any new information. 
The random play feature provides that when the machine is in 
trouble, it tries various plays until it "accidentally" plays to 
the total in its memory where the trouble is, and proceeds to 
correct it. On the next game, it then uses this good information 


to beat the human, 


To provide variety and to keep interest up when the machine has 
completely learned the game, I have provided a “goof” generator. 
This is an assymmetric flip-flop that drives relay 26. At a 
particular point in the program, just before the machine play 
selector 4 starts its search for a winning move, the programmer 
"clamps" this flip-flop, which has about a 6 to 1 duty cycle. 
Thus, about one out of 7 "clamps" results in relay 26 operating. 
When this occurs, relay 20 is activated causing the machine to 
immediately select a move at random instead of searching for a 
winning move. Of course, the randomly selected move may be a 
winning move and the human then does not benefit. If it is a 
losing move and the human recognizes this, it gives him a chance 


to seize the winning advantage and win the game, provided he does 
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not make any mistakes and let the machine regain the advantage. 


The random play generator control relay 20 is returned to its 
unclamped condition at the end of each cycle of turns by a contact 


on relay 17. 


Let us turn now to the modus operandi of the memory scanner. It 
comprises a truck that moves back and forth over the memory array ~ 
which in turn consists of 17 columns and 6 rows of neon lamps. 
There is no column corresponding to total 0, because the machine 
never has to play from 0, nor to eoiuad 18, because there is ob- 
viously no winning move at that total. The position of the 
scanner is controlled by a sexvo system that functions as follows: 
(the schematic of this servo amplifier is not shown on the main 
diagram, but is shown in Figure 6) Totalizer 2, using two sets 

of bank contacts, causes an adjacent pair of incandescent lamps 

to be lighted in a 7th row along one side of the memory display. 
These two lamps are driven by alternate half cycles of the 60 cps 
supply voltage, consequently their light outputs contain oppositely 
phased 60 cps components. <A CdS photocell is mounted on the 
scanner so that it "looks at" this row of lamps, and the AC 
component of the photocell output is amplified, converted to DC 

by a phase sensitive rectifier that is driven by the 60 cps line 
voltage, further amplified, and applied to a DC motor that drives 
the scanner. A 60 cps AC slewing voltage is connected through a 
diode to the input of the servo amplifier, and the DC component of 
the photocell output is used to determine whether this slewing 
voltage is applied to the input or not. This arrangement may be 


novel - its advantage is that if the photocell sees more than a 
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certain amount of light, the slewing voltage is effectively 
disconnected from the input of the servo and does not cause any 
offset from the null position of the servo where the two 60 cps com- 
ponents of the light from the incandescent lamps balance. When 
the scanner is far from the null position, the average light input 
to the photocell decreases greatly, the DC output of the photocell 
decreases, and the diode is biased into conduction ~ connecting in 
the slewing voltage. The slewing voltage is required because 

the totalizer stepper 2 advances much faster than the servo can 
follow. During the course of the game the slewing voltage is 
phased to drive the scanner towards the end of the game portion of 
the memory; when the NEW GAME push button is operated, the slewing 
voltage phase. is reversed, driving the scanner back towards the 


beginning of the game. 


To provide damping of the servo so that the system is stable and 

does not hunt, the motor is driven in a bridge circuit, the 

output of which is a voltage proportional to motor velocity. This 
drives a transistor amplifier whose output is combined with the output 
of the phase sensitive rectifier thus effectively providing velocity 


damping for the servo. 


It is necessary to operate a relay, 19, whenever the motor is 
running. The motor voltage is fed to two biased off transistor 
amplifiers, One of these feeds directly, and the other feeds through 
a phase or polarity inverting stage, a transistor that drives relay 
19. The input circuit of this final stage incorporates an RC 

network that gives this circuit a quick-attack, slow-release charac- 


teristic, which permits the motor to come to a complete stop before 
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the programmer is signalled to continue. 


The scanner truck carries 6 photocells, each of which "sees" the 
light from any one of four neons in a column of the memory. 

Figure 5 shows a representation of the memory scanner positioned 

for two different totals, 11 and 13. The holes cut in the card- 
board slider depict the fields of view of the 6 photocells, each of 
which looks at a vertical column of holes. The first photocell looks 
at the column that results from the addition of 1 to the total. 

In Figure 5, where the position of the scanner corresponds to a 
present total of 11, the first photocell sees column 12, the second - 
column 13, etc. The first photocell can see only rows 2, 3, 4, and 

5 - as these correspond to the legal responses that the human can 
choose from if the machine were to play al. Similarly, the second 
photocell, which searches for winning moves in the column that 
corresponds to the addition of 2 to the total, can see only those 
winning moves that would be legal responses to a machine move of 2, 
namely, 1, 3, 4, and 6. Similarly the other 4 photocells look 

ahead in the memory. A permanently lighted patch that includes 

rows 2 and 3 (moves that cannot both be illegal) and columns 19 
through 24, tells the scanner that any move that takes the total 


beyond the end of the game is a losing move. 


In Figure 5, the scanner is positioned as it would be if the total 
were 13. The winning play 4 in column 14 is seen by the first 
photocell, and results in the absence of a lit neon in row 1, column 
13. If this neon were lit, the machine would turn it off at this 
point. Similarly, lit neons visible in columns 15, 16, and 17 


result in unlit neons in column 13, rows 2, 3, and 4. Column 18 has 


es: 2 
no neons mounted in it, as there is obviously no winning move when 
the total is 18. Thus photocell 5 sees no light, the machine 
concludes that there is no winning move available to its opponent 
if it plays 5, accordingly it lights the neon in column 13, row 5. 
If this move is legal the machine will make such a move. If not, 
it will continue to photocell 6, which sees a portion of the perma~ 
nently lighted patch beyond the end of the game, resulting in an 


unlit neon at column 13, row 6. 


Each photocell is connected to a bank contact on the machine play 
selector 4. As this selector steps through its cycle, the photocells 
are energized one at a time in sequence. A photocell that sees 

the light of one or more neons conducts and biases the 6AU6 vacuum 
tube into conduction. The plate circuit of this tube drives a transis~ 
tor that in turn operates relay 6. At the same time that each photo- 
cell is energized, selector 4 applies a common arm of the form C 
contact of 6, through the wipers and bank contacts of totalizer 2 

to the appropriate memory cell. If 6 is operated, corresponding 

to indication that the move being considered is a losing move, a 
pulse of voltage is applied to the memory cell that tends to turn 
off the neon in the memory array, and turns on the corresponsing 

neon in the panel display. If 6 is not operated a pulse of the 
oppesite polarity is applied, which turns on the appropriate neon 

in the memory array and turns off the corresponding one in the memory 


front panel display. 


At the outset of a sequence of games, the memory can be set to a 
complete blank (All neons extinguised in the internal array; all 


panel memory indicators lit) by operation of the MEMORY RESET push 
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button S16. If desired, to speed the learning process, the 6 
winning moves needed to reach 18 from totals 17, 16, 15, 14, 13, 

and 12, can be set in by operation of the MEMORY PRELOAD push button, 
815. It will be appreciated that at the outset of a sequence of 
games, when the memory has been reset, the scanner, seeing no 

lit neons, thinks that all moves are winning. It thus lights a 

good many neons, some of which actually are winning moves and some 
of which are actually losing moves. These latter correspond to 
"mistakes". They cause the machine to play incorrectly, as is 

to be expected. Gradually, as more and more games are played, 

"good" information, the blank column at total 18, and the lighted 
patch in columns 19 through 24, propagates backwards through the 
memory till nearly all mistakes are corrected and nearly all winning 
moves;are found. A few mistakes and winning moyen undiscovered 

are unimportant to the machine winning games, because at some totals, 
there are many winning moves, and the machine tends to play only the 
smallest legal one. For example, at total 11, 2,3,4, and 6 are 
winning moves. The machine does not need to know more than those 

at 2 or 3, as one is bound to be a legal winner and the machine 
searches in the sequence, 1,2,3,%,*,*,. It does need to find the 
rest so that its play at earlier totals will be such as to prevent 
the human from getting to 11. However, there are some such totals 
with many winning moves close to the beginning of the game and it is 


the winning moves at these points that the machine tends to ignore. 


It was necessary to permanently blank one point in the memory, 
coiumn 5, row 2. This was necessary to prevent the following 


situation from occurring. On the first game, the human may play | 
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1's whenver it is his turn (ti11 near the end of the game, when 

he plays to win) and the machine will respond with 2's, the lowest 
legal reply, which it chooses since it has no knowledge of winning. 
moves, cannot see that any are available to its opponent, and conse- 
quently thinks that 2 is a winning move. During this first game a 
mistake is recorded in column 7. This mistake cuuses the machine, 
in subsequent games, to erroneousiy mark down a "winning" move at 
row 2, column 5. This mistake, in turn, prevents the machine from 
finding the only correct reply to an opening move of 1 by the human - 
4. Further, if the human opens with 1, the machine cannot play a 6, 
which would otherwise eventually result in correction of the mistake 
at column 7. Furthermore, correct play by the human will result in 
column 7 never being examined by the machine. This dilemma can be 
solved only by arranging that the neon in the memory at column 5, 


row 2 cannot be turned on. It could have been simply omitted. 


A sequence of 15 to 20 games is sufficient for the machine to learn 
the game nearly completely, if the human plays skillfully. I have 
noted that if the human does not play correctly, the machine learns 
more Slowly; correct jutormatven is generated near the end of the 
game, where the machine needs it, but propagates very slowly towards 


the beginning. 


The "goof" generator occasionally results in an amusing situation. 
Once a friend, in playing his first game, was defeated by the machine 
because of a "sheer luck" sequence of "goofs" in which the machine 


accidentally chose a sequence of winning moves! 
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Playing "18-with-a-die" 


' Confronted with a situation, the player says to himself: 
I have four possible plays.: Which is best? 
I want the play that einge me to 18 with the shortest run. 
If a good play is available, it will still be available after my opponent has 
played, and the 7~complement too. 
Let us say my score is 3 and it is my turn to play. 
- if J play 5, 1 att have a sure run of 4 plays to 18: 
3 § 13 18° 
. If I play 3, I have a sure run of 5 plays to 18 (4 is also available) 
3 6-10 14 18 
If I play 1 or 6, I ee I will have a long run. [ will avoid tunis play. 
Now, if 5 is available, it is my best play. If not, 3 is surely available; 
it is my second-best play. 


The HUMAN player hes to FIGURE OUT or REMEMBER , for each situation, which number 
is F=first choice and which number is S=second choice. 


If he plays the best choice every acs he is sure to win (barring a run of bad luck, ) 
aceinet an opponent who does not figure or remember. 


Now, figuring is long and not always reliable, and memory is short. 


If the opponent is the machine, it can be supplied with the BEST PLAYS for every 
situation end will deliver the correct M-play every time. 


If I had a computer, it would supply me with a complete list of all possible 
vleys to 18 for every situation. From this list, I could easily pick out the BEST PLAY, 
First choice and Second choice, for every Mepoore and every Human—play. 


Now I have here a small computer which was given to me when I was born. 
It came out with a DICTIONARY OF BEST PLAYS, which is implemented in the MEMORY. 
— Here is how the Machine works: 
I. Between plays, it displays its actual score: M-Score. 
It displays its last M-Play, so the player knows the numbers available to him. 
The counting oecillator is running and the counting is going on in the four 
Rendcom-Play counters. 

II. HUMAN PLAYER makes nis choice of a play by pushing one of six switches, 
avoiding the plays inhibited by the M-Play. (Should be loud signal if he goofs.) 

III. HUMAN PLAYER presses the PLAY button. PROGRAM starts. Random counting STOPS. 

Imneciately, the next M-Play is available on the l-Pley line, 
either F or S or Random, according to the situation. 
At Tl, the registers are cleared. 
At Te, M-Play is entered in the M-Play Register activating the new display, 
end in the M-Play.Adding Register. 

At 13, M-Play is added to previous M-Score, and ectual M-Score is displayed. 


At STOP Signal, Program STOPS, and Counting Oscillator STARTS. 


NOTE: The RANDOM-PLAY LOGIC tekes care of the situation: . 
1° If M-Score is 0 and HChoice is 0, M-Play will be Random 1-2-3-4—5-6 
2° If M-Score is 0 and H-Choice #0, M-Play will be one of the 

four numbers allowed. 
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I use diodes to isolated the 40 possible outputs to the M-Pley lines. 
I think they are competent. 


Could be gates? Yes, but with annoying inversion, cumbersome wiring. 
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M-score is decoded and inverted by NAND gates 
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Oc 52/1 0] The Oscillator is the one used to totalize M-Score 

Tt runs continuously between plays, when Program is Off. 
It is blocked when Program is ON, offering ONE N-Play 
either 1-2-3-4-5-6 or 1-2-3-4, according to the situation. 
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